Multimodal Grammar Implementation
نویسندگان
چکیده
This paper reports on an implementation of a multimodal grammar of speech and co-speech gesture within the LKB/PET grammar engineering environment. The implementation extends the English Resource Grammar (ERG, Flickinger (2000)) with HPSG types and rules that capture the form of the linguistic signal, the form of the gestural signal and their relative timing to constrain the meaning of the multimodal action. The grammar yields a single parse tree that integrates the spoken and gestural modality thereby drawing on standard semantic composition techniques to derive the multimodal meaning representation. Using the current machinery, the main challenge for the grammar engineer is the nonlinear input: the modalities can overlap temporally. We capture this by identical speech and gesture token edges. Further, the semantic contribution of gestures is encoded by lexical rules transforming a speech phrase into a multimodal entity of conjoined spoken and gestural semantics.
منابع مشابه
Implementing Tabular Deduction for Multimodal Logical Grammar
The paper proposes a method for tabular deduction for multimodal logical grammar, extending a method for linear logic presented in (Hepple, 1996; Hepple, 1998). Tabular deduction for multimodal logical grammar avoids redundant computation like chart parsing does for CFGs. The method discussed here enables one to process any kind of multimodal logical grammar fragment. An implementation of the m...
متن کاملReasoning with Categorial Grammar Logic
The article presents the first results we have obtained studying natural reasoning from a proof-theoretic perspective. In particular we focus our attention on monotonic reasoning. Our system consists of two parts: (i) A Formal Grammar – a multimodal version of classical Categorial Grammar – which while syntactically analysing linguistic expressions given as input, computes semantic information ...
متن کاملHybrid Planning and Realization of Coherent Utterances for Multimodal Natural Language Dialogue Systems
The output of multimodal human-computer interfaces is what this thesis is concerned with. Rather than hard-coding graphical and spoken representations, methods are introduced that plan and realize coherent output, appropriate to the situation and the device. The generation system expects a modeand language-independent representation, as it can be supplied by the dialogue management component of...
متن کاملA Categorial Type Logic
In logical categorial grammar (Morrill 2011[22], Moot and Retoré 2012[10]) syntactic structures are categorial proofs and semantic structures are intuitionistic proofs, and the syntax-semantics interface comprises a homomorphism from syntactic proofs to semantic proofs. Thereby, logical categorial grammar embodies in a pure logical form the principles of compositionality, lexicalism, and parsin...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2012